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Surveys  are  the  most  popular  method  for  researching  human  behavior  in  organizations 
(Church,  2001).  Most  of  what  we  know  about  equal  opportunity  (EO)  perceptions,  climates,  and 
behaviors  has  been  gleaned  from  EO  surveys  (Knouse  &  Dansby,  2000).  On  the  positive  side, 
surveys  provide  standardized  formats  that  allow  statistical  analyses  of  responses  according  to 
demographic  data,  such  as  gender,  race,  ethnicity,  and  age.  Surveys  are  the  most  cost  effective 
method  of  research;  i.e.,  the  cost  per  respondent  is  lower  than  that  for  natural  observation  and 
experimentation  with  all  their  equipment  and  manpower  needs.  Most  importantly,  surveys  allow 
for  the  possibility  of  a  large  number  of  respondents  (Church  &  Waclawski,  2001 ;  Rosenfeld, 
Edwards,  &  Thomas,  1993). 

The  major  negative  point  of  surveys  is  that  return  rates,  defined  as  the  number  of  usable 
surveys  returned  divided  by  the  total  sample  originally  receiving  the  survey  (Edwards,  Thomas, 
Rosenfeld,  &  Booth-Kewley,  1997),  tend  to  be  low.  As  a  traditional  rule  of  thumb,  a  response 
rate  of  over  50%  is  considered  theoretically  adequate,  while  above  60%  is  good  (Babbie,  1973). 
In  reality,  the  rates  tend  to  be  lower.  In  the  civilian  world,  overall  survey  return  rates  range  from 
10%  to  50%  (Dillman,  2000).  Meta-analyses  of  research  studies  of  return  rates  show  average 
rates  of  46%  to  49%  (Church,  1993;  Heberlein  &  Baumgartner,  1978),  which  according  to 
Babbie  is  just  at  the  adequate  threshold.  More  specifically,  mail  surveys  of  corporations  tend  to 
produce  return  rates  of  20-30%  (Henderson,  1990).  A  study  of  the  leading  journals  in 
management  and  the  behavioral  sciences  over  a  period  of  more  than  30  years  showed  an  average 
return  rate  of  55.6%  with  a  standard  deviation  of  19.7,  median  of  60%  and  mode  of  45%  for 
questionnaire-based  research  of  organizations.  Very  significantly,  the  return  rate  declined  over 
time  (Baruch,  1999).  Indeed,  return  rates  in  all  classes  of  surveys  have  decreased  over  time 
(Krosnick,  1999),  including  e-mail  surveys  (Sheehan,  2001).  In  addition,  a  study  of  Web-based 
electronic  surveys  showed  a  return  rate  of  39.6%  -  lower  than  averages  for  paper  and  pencil 
questionnaires  (Cook,  Heath,  &  Thompson,  2000).  In  sum,  the  real  return  rates  are  not 
impressive. 

Today  surveyors  face  a  basic  dilemma.  On  the  one  hand,  organizations  are  increasingly 
establishing  goals  of  high  return  rates,  such  as  80%,  in  order  to  justify  the  time  and  cost  of 
surveying.  On  the  other  hand,  institutional  research  review  boards  are  emphasizing  the  voluntary 
nature  of  surveys;  respondents  do  not  have  to  answer,  if  they  do  not  wish  to  do  so,  which  can 
translate  into  many  incomplete  surveys  and  significantly  lower  return  rates  (Dillman,  2000). 


The  military  tends  to  have  higher  return  rates,  ranging  from  30%  to  70%,  although  the 
rates  tend  to  be  decreasing  for  a  variety  of  reasons  (Rosenfeld,  Newell,  Harris,  &  Hindelang, 
2002),  as  we  shall  see.  Focusing  upon  military  equal  opportunity  (EO)  surveys,  several  recent 
studies  show  a  range  of  return  rates  from  41%  to  76%  (see  Table  1).  The  average  return  rate  for 
the  surveys  described  in  Table  1  is  53%  (with  a  standard  deviation  of  1 1.72).  This  is  respectable 
but  could  be  better,  particularly  considering  the  large  variability  in  rates  and  a  historically  higher 
rate  of  return  (Rosenfeld  et  al.,  2002). 

The  present  report  examines  survey  return  rates  and  how  they  can  be  improved.  First,  I 
present  a  review  of  the  civilian  literature  on  surveying.  Then,  I  look  at  survey  administration  in 
the  military.  From  these  reviews,  I  present  a  number  of  recommendations  for  improving  the 
return  rate  of  EO  surveys. 
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Table  1 

Return  Rates  of  Recent  Military  Equal  Opportunity  Surveys 


Source 

Survey 

No. 

Items 

Purpose 

Time  Period 

Sample  Size 

Return 

Rate 

MEOCS 

Service  Report 

MEOCS 

124 

Evaluate  EO 
climate 

Apr01-May02 

482,264 

US  military 

40% 

Stewart  (2000) 

Armed 

Forces  EO 
Survey 

81 

Evaluate  EO 
perceptions 

Sep96-Feb97 

76,754  US 
military 

53% 

Davis  & 

Perron  (2001) 

Mixed 

Gender 

Opinion 

Quest. 

19 

Evaluate  women’s 
issues 

May97 

2,460 

Canadian 

Forces 

62% 

Estrada  et  al. 
(2001) 

Sexual 

Experience 

Quest. 

23 

Evaluate  sexual 

harassment 

incidents 

Mar-Apr99 

424  Swedish 
female 

officers/cadets 

76% 

Barnes  (2001) 

MEOCS 

Evaluation 

Survey 

14 

User  evaluation  of 
MEOCS 

May96-Apr97 

375 

commanders 

375 

administrators 

41% 

37% 

Dansby 

(1998a) 

SLEOCS 

101 

Evaluate  EO 
climate 

Mar95-Nov97 

1,947  flag 
officers 

66% 

Dansby 

(1998b) 

MEOCS- 

LITE 

94 

Evaluate  EO 
climate 

1996 

8,800  Air 

Force  military 

45% 

Edwards 

(1998) 

Sexual 

Harassment 

Surveys 

Form  A 

Form  B 

56 

132 

Evaluate  sexual 

harassment 

incidents 

A:  old  survey 

B:  new  survey 

1995 

DoD  active 
duty 

A:  30,239 

B:  49,752 

46% 

58% 

Johnson 

(1996) 

Supervisor 

Evaluation 

of 

Graduates 

20 

Evaluate  DEOMI 
graduates 

1994-1995 

111 

graduates’ 

supervisors 

57% 

Review  of  the  Civilian  Literature  on  Survey  Return  Rates 


There  are  a  large  number  of  studies  in  the  civilian  literature  highlighting  factors 
influencing  survey  return  rates  and  how  to  increase  these  rates. 

Length 


One  of  the  factors  examined  extensively  is  the  length  of  the  survey  itself.  Overall,  long 
surveys  tend  to  produce  lower  response  rates  (Edwards  &  Thomas,  1993).  The  concept  of 
length,  however,  is  more  complex  than  simply  item  or  page  counts.  Shortening  already  short 
surveys  does  not  improve  the  return  rate.  Similarly,  cramming  more  questions  into  a  fewer 
number  of  pages  does  not  work,  nor  does  shrinking  font  size  (Dillman,  2000).  The  key  point 
appears  to  be  to  create  a  long  enough  survey  that  the  respondents  understand  the  questions  and 
have  sufficient  space  to  respond,  while  at  the  same  time  make  it  short  enough  that  the 
respondents  do  not  feel  imposed  upon. 

A  corollary  problem  to  return  rate  is  response  rate;  i.e.,  how  many  of  the  items  in  a 
survey  do  respondents  complete.  Length  is  also  a  factor  here.  In  a  recent  study  of  Federal 
government  employees,  Church  (2001)  found  that  longer  surveys  (150  and  129  items)  had  a 
significantly  higher  skip  rate  (more  items  unanswered)  than  a  shorter  survey  of  58  items.  Church 
speculates  that  a  fatigue  factor  was  present;  respondents  simply  got  tired  of  answering  items  in 
the  longer  surveys. 

Questionnaire  Format 

Generally,  surveys  incorporate  a  multiple  choice  or  combination  multiple  choice  and 
open-ended  question  format  (Dillman,  2000).  Surveys  composed  almost  entirely  of  open-ended 
questions,  however,  tend  to  exhibit  low  return  rates  (22%)  (Falik  &  Carroll,  1971),  although  new 
data  show  that  open-ended  questions  do  not  exert  as  negative  an  impact  as  once  thought 
(Krosnick,  1999).  Questions  of  a  sensitive  nature,  particularly  about  finances,  tend  to  decrease 
return  rate,  unless  guarantees  of  anonymity  are  given,  which  can  improve  the  return  rate  up  to 
nine  percentage  points  (Mitchell,  1998). 


With  the  increase  in  computer  usage,  more  computer-based  surveys  are  being 
administered.  The  first  type  was  e-mail  surveys  in  the  mid-1980s  that  are  now  dramatically 
decreasing  in  frequency.  Apparently,  Web-based  surveys  with  more  flexibility  and  more  options 
are  replacing  e-mailed  surveys.  In  addition,  different  e-mail  software  varies  widely  in  what  can 
be  presented  on  the  computer  screen.  Thus,  e-mail  offers  fewer  options  than  Web-based  surveys 
for  potentially  increasing  return  rate  through  attracting  the  attention  of  the  computer  user.  These 
factors  include  variables  that  may  enhance  the  look  of  the  survey  like  color,  response  formatting, 
and  imaging  (Sheehan,  2001). 

Pre-notification 

Some  type  of  pre-notification,  such  as  a  letter  or  e-mail,  can  let  the  individual  know  that 
an  important  survey  is  coming.  Psychologically,  this  can  build  anticipation  in  the  individual 
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(Dillman,  2000).  A  review  of  the  response  rate  literature  estimated  that  pre-notification  by 
phone  increases  return  rate  16%,  by  letter  6%,  and  by  postcard  2.5%  (Haggett  &  Mitchell,  1994). 
In  addition,  pre-notification  increases  response  speed  in  both  mail  surveys  and  e-mail  surveys 
(Sheehan,  2001). 

One  promising  area  of  pre-notification  is  by  e-mail,  which  can  reach  a  large  sample 
quickly.  The  problem  is  that  individuals  are  being  inundated  with  unsolicited  e-mail  messages, 
which  may  lead  people  to  ignore  e-mail  survey  pre-notifications  (Sheehan,  2001). 

Follow-Ups 

Follow-ups  in  the  form  of  a  second  survey  mailed  or  a  phone  or  postcard  follow-up  can 
increase  return  rates.  In  a  meta-analysis  of  survey  studies,  Heberlein  &  Baumgartner  (1978) 
found  that  a  follow-up  increased  average  return  rates  from  46%  to  68%  --  a  sizeable  increase. 
One  review  of  the  literature  estimated  a  7%  increase  for  a  postcard  follow-up,  16%  for  a  letter 
follow-up,  and  21%  for  a  letter  accompanied  by  a  copy  of  the  survey  (Mitchell,  1998).  Certified 
mail  follow-ups  have  increased  rates  by  up  to  30%.  The  problem  is  that  unless  the  respondent  is 
present,  he  or  she  must  go  to  a  post  office  to  sign  for  the  certified  letter,  which  may  increase 
irritation  on  the  part  of  the  respondent  to  the  point  of  refusing  to  complete  the  survey  (Dillman, 
2000). 


In  the  relatively  new  area  of  Web-based  electronic  surveys,  some  assert  that  follow-up 
notices  can  double  response  rates  (Kittleson,  1995).  With  e-mail  surveys,  the  response  rate  can 
increase  25%  (Sheehan,  2001).  The  problem  is  that  these  contacts  are  typically  in  the  form  of  e- 
mail  messages.  The  potential  exists  for  a  large  number  of  e-mail  contacts  that  can  be  directly 
transmitted.  The  respondent,  however,  can  easily  become  overwhelmed  by  this  large  number  of 
reminders.  A  review  of  Web-based  surveys  showed  that  as  the  number  of  follow-ups  increased, 
return  rates  actually  decreased  (Cook  et  al.,  2000). 

Incentives 

Incentives  are  among  the  most  effective  devices  for  increasing  return  rates  (Mitchell, 
1998).  Theoretically,  including  something  of  value  in  the  survey  package,  like  money  or 
coupons  or  even  a  decal,  creates  a  psychological  obligation  in  the  respondent,  which  should  lead 
to  reciprocation  by  the  respondent  (Arzheimer  &  Klein,  1999).  In  essence,  the  respondent 
perceives  that  the  surveyor  has  entrusted  an  item  of  value  to  the  respondent  in  the  survey 
package  before  the  survey  is  even  returned.  Therefore,  the  respondent  feels  obligated  to 
reciprocate  by  completing  and  returning  the  survey. 

In  a  meta-analysis  of  survey  research,  Church  (1993)  found  that  the  return  rate  for 
surveys  without  incentives  was  28%,  which  was  increased  19  %  when  financial  incentives  were 
included  in  the  survey  package.  Similarly,  a  meta-analysis  by  Yammarino,  Skinner,  &  Childers 
(1991)  showed  up  to  a  25%  increase  with  various  types  of  incentives.  Typically,  financial 
incentives  occur  in  the  form  of  one-dollar  bills,  although  sometimes-higher  denominations  are 
used.  Coins  occur  occasionally,  but  are  difficult  to  handle  in  survey  packages.  Checks  for  one 
dollar  are  deemed  by  many  respondents  as  too  much  trouble  to  cash  (Dillman,  2000). 
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The  Church  (1993)  meta-analysis  also  found  that  nonfinancial  material  incentives 
increased  return  rates  8%.  Ballpoint  pens  are  a  typical  type  of  material  incentive;  although  it 
may  be  that  the  unique  packaging  used  to  encase  a  ballpoint  pen  is  what  attracts  the  respondent 
rather  than  the  pen  itself  (Dillman,  2000).  Lottery  tickets  are  gaining  frequency  as  another 
incentive.  The  evidence  of  their  effectiveness  is  marginal,  however  (Dillman,  2000).  On  the 
other  hand,  phone  cards  are  gaining  prominence  as  incentives,  because  they  are  perceived  as  both 
financial  and  material  incentives.  In  a  German  study,  a  six  Deutsch  mark  phone  card  (about 
$3.60  U.S.)  raised  return  rates  12%  or  the  equivalent  of  using  a  U.S.  $5  bill  (Arzheimer  &  Klein, 
1999). 


Incentives  promised  after  returning  the  survey  are  not  effective.  Apparently  the 
respondent  does  not  believe  they  will  come,  or  they  prefer  to  see  the  incentive  up  front  when 
they  open  the  survey  package  (Dillman,  2000).  In  addition,  frequent  organizational  surveys  may 
decrease  the  value  of  incentives.  Employees  may  feel  that  each  subsequent  survey  should  have  a 
larger  incentive  to  make  it  worth  their  time  (Edwards  et  al.,  1997). 

Promising  to  send  the  respondent  a  summary  of  the  survey  results  also  does  not  increase 
response  rate.  A  possible  reason  is  that  respondents  may  believe  that  their  anonymity  would  be 
compromised  by  sending  their  address  in  order  to  receive  the  results  (Mitchell,  1998). 

Interestingly,  the  use  of  incentives  with  Web-based  surveys  tends  to  decrease  the  return 
rate.  Some  speculate  that  many  Web  surveys  are  too  long  or  require  so  much  effort  to  complete 
that  respondents  expect  substantial  incentives  for  working  through  the  Web-based  survey  (Cook 
et  al.,  2000).  One  promising  area  in  Web-based  incentives  is  the  accumulation  of  points  for 
completing  surveys  that  may  be  exchanged  for  merchandise  at  other  Web  sites  (Sheehan,  2001). 

Postage 

A  meta-analysis  of  type  of  postage  affixed  to  the  survey  package  and  return  envelope 
found  that  first-class  stamps  were  significantly  better  than  bulk  and  business  postage  (Armstrong 
&  Lusk,  1987).  One  explanation  is  that  real  stamps,  particularly  commemorative  stamps  in 
several  colors,  are  more  apt  to  catch  the  eye  of  the  individual  than  bulk  postage  (Dillman,  2000), 
which  may  make  the  survey  package  look  like  junk  mail.  Once  the  stamp  has  caught  the 
individual’s  eye,  he  or  she  may  be  more  apt  to  complete  the  survey,  because  they  believe  that 
investing  in  more  expensive  first-class  postage  means  that  the  survey  is  important  enough  to 
justify  that  cost. 

Sponsorship 

There  is  some  evidence  that  type  of  sponsorship  of  the  survey  may  make  a  difference. 

One  study  showed  that  subjects  were  more  apt  to  open  a  survey  introduced  in  the  cover  letter 
with  official  government  sponsorship  (U.S.  Census  Bureau)  than  a  mass  marketing  sponsorship 
approach  (Dillman,  Jenkins,  Martin,  &  DeMaio,  1996).  University  sponsorship  can  increase 
return  rate  even  more  than  government  sponsorship  (Mitchell,  1998).  On  the  other  hand,  a  meta¬ 
analysis  showed  that  overall  sponsorship  did  not  significantly  affect  return  rates  (Y ammarino  et 
al.,  1991).  Moreover,  a  plea  to  help  the  sponsor  combined  with  incentives  for  completing  the 
survey  actually  tends  to  decrease  the  return  rate  (Mitchell,  1998). 
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Survey  Salience 

Survey  salience  involves  how  well  the  survey  deals  with  issues  that  are  relevant  and 
meaningful  to  respondents  (Edwards  et  al.,  1997).  In  a  review  of  the  survey  return  literature 
Heberlein  and  Baumgartner  (1978)  found  an  incredible  77%  return  rate  for  salient  surveys 
compared  to  a  42%  rate  for  nonsalient  surveys.  They  believe  that  survey  salience  is  the  strongest 
factor  in  return  rates.  Nonsalient  surveys  may  be  viewed  by  respondents  as  a  waste  of  time  or 
may  reflect  a  lack  of  enthusiasm  for  the  value  of  social  science  surveying.  The  respondent  may 
ask  how  the  survey  benefits  him  or  her  directly  (Baruch,  1999).  The  pre-notification  letter  or 
survey  cover  letter  can  present  the  purpose  of  the  survey  in  terms  of  its  importance  to  the 
respondent  (Fowler,  1993). 

Edwards  et  al.  (1997)  recommend  increasing  survey  salience  by  adding  items  of  specific 
interest  to  respondents  and  placing  at  least  a  few  of  these  items  toward  the  front  of  the  survey  to 
generate  interest  as  respondents  begin  the  survey.  Dillman  (2000)  recommends  adding  items 
asking  about  current  behaviors  and  interests  to  increase  salience.  Of  course,  there  can  be  a 
mitigating  effect  with  survey  length.  Adding  too  many  items  to  increase  salience  may  also 
increase  survey  length,  which  may  partially  cancel  out  the  salience  advantage. 

Personalization 

With  the  advent  of  computer  word  processing  and  high-speed  printers,  cover  letters  can 
be  highly  personalized  by  including  the  respondent’s  name  and  address  and  embedding  personal 
information  in  the  cover  letter.  Research,  however,  shows  mixed  results.  Some  studies  find  that 
such  personalization  heightens  the  perceived  salience  of  the  survey,  while  others  believe  that 
personalization  may  cause  respondents  to  question  whether  the  survey  is  truly  anonymous  and 
whether  confidentiality  of  their  responses  will  truly  be  secure  (Dillman,  2000;  Mitchell,  1998; 
Edwards  et  al.,  1997). 

Survey  Features  that  Do  Not  Increase  Return  Rate 

An  obvious  variable  that  would  seem  intuitively  to  work  because  it  would  attract  the 
reader’s  attention  is  color  of  the  survey.  Reviews  of  the  literature  uniformly  find  that  color  has 
no  effect  (Dillman,  2000;  Mitchell,  1998).  Another  variable  that  would  seem  to  be  effective  is 
prescribing  deadlines  by  which  the  surveys  must  be  returned.  Again  reviews  of  the  literature 
find  no  effect  (Mitchell,  1998;  Edwards  et  al.,  1997). 

Demographic  Influences 

Several  respondent  background  variables  may  affect  survey  return  rate. 

Age.  Choice  of  survey  format  may  relate  to  survey  return  rates,  and  age  may  be  a  strong 
factor  in  this  choice.  The  argument  here  is  that  respondents  are  more  apt  to  return  those  surveys 
possessing  formats  that  they  prefer  to  use  in  completing  the  survey.  Younger  respondents  tend  to 
choose  Web-based  survey  formats  over  more  traditional  forms,  such  as  paper  and  pencil,  which 
are  preferred  by  older  respondents.  This  finding  tends  to  cut  across  education  level.  Younger 
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doctors  preferred  the  Web-based  format,  while  older  doctors  preferred  the  more  traditional 
approaches  (Church,  2001).  Moreover,  individuals  with  higher  computer  anxiety,  such  as  older 
respondents,  produced  lower  quality  answers  (more  errors,  less  apt  to  finish  the  survey,  lower 
cognitive  performance)  (Booth-Kewley,  Rosenfeld,  &  Edwards,  1993). 

Gender.  In  the  area  of  choice  of  survey  format,  there  were  no  differences  between  males 
and  females  in  choosing  Web-based  or  more  traditional  survey  techniques  (Church,  2001).  In  an 
all-female  sample,  women  initially  returned  the  survey  at  a  rate  of  58%,  while  a  follow-up  phone 
message  increased  the  rate  to  66%  (Brennan  &  Hoek,  1992). 

Education  Level.  In  general,  respondents  with  higher  education  levels  (college  degree 
and  higher)  return  surveys  at  a  higher  rate  than  less  educated  respondents.  This  can  introduce  a 
strong  response  bias  into  the  survey  data.  The  findings  may  be  skewed  in  favor  of  the  responses 
of  the  more  highly  educated  and  literate  members  of  the  organization  (Edwards  et  al.,  1997). 

Personality.  Although  the  personality  of  respondents  and  nonrespondents  has  not  been 
extensively  studied,  there  are  some  findings  about  the  type  of  person  who  responds  to  survey 
requests.  He  or  she  tends  to  reciprocate,  to  follow  commitments,  to  follow  the  lead  of  others 
similar  to  himself  or  herself,  to  follow  suggestions  of  authority  figures,  to  seize  unique 
opportunities,  and  to  favor  others  he  or  she  likes  (Groves,  Cialdini,  &  Couper,  1992).  Moreover, 
respondents  are  more  likely  to  believe  that  it  is  their  social  responsibility  to  complete  surveys, 
that  they  can  influence  public  policy,  and  that  they  are  generally  happy  with  their  lives. 
Conversely,  nonrespondents  tend  to  live  in  large  cities  and  work  long  hours  (Krosnick,  1999). 

Culture.  Culture  as  well  as  country  of  residence  may  influence  which  survey  format 
respondents  choose.  In  a  multinational  surveying  effort,  Americans  preferred  Web-based  and 
automated  phone  techniques,  while  South  Americans,  Europeans,  Africans,  and  Asians  preferred 
more  traditional  survey  methods.  This  finding  was  due  in  part  to  the  lower  quality  phone  and 
computer  connections  in  several  countries  (Church,  2001). 

In  addition,  respondents  from  different  cultures  may  be  less  apt  to  respond  to  surveys, 
which  violate  cultural  norms,  such  as  describing  personal  information,  or  evaluations,  which  tend 
to  pit  colleagues  against  one  another  (Church,  2001).  Indeed,  a  meta-analysis  of  multinational 
survey  return  rates  showed  that  American-style  surveys  had  low  return  rates  with  high  variability 
(high  standard  deviation)  in  non-Westem  countries  (Baruch,  1999).  The  author  cautions  that  the 
small  number  of  studies  in  this  meta-analysis  sample  makes  the  conclusions  tentative. 

Type  of  Employee.  In  a  study  of  organizational  research  surveys,  type  of  employee 
influenced  return  rate.  Managers  returned  surveys  the  most  (62%)  closely  followed  by 
employees  (61%)  and  professionals  (59%).  Top  managers  including  chief  executive  officers 
(CEOs)  had  the  worst  return  rates  (36%)  (Baruch,  1999).  In  the  area  of  academics,  physical 
scientists  had  the  worst  return  rates  (30%),  while  social  scientists  have  the  highest  rates  (63%) 
(Mitchell,  1998).  One  explanation  is  that  social  scientists  are  very  familiar  with  surveying  since 
they  use  it  as  a  primary  research  device,  while  physical  scientists  see  surveying  as  “soft”  science 
whose  data  are  too  qualitative. 
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The  Questionable  Validity  of  Very  High  Returns 

The  prevailing  wisdom  has  been  to  maximize  return  rates  toward  goals  of  70%  or  even 
greater  (Dillman,  2000).  New  evidence,  however,  shows  that  very  high  rates  are  not  necessarily 
accurate.  Indeed,  reporting  rates  of  90%  and  above  should  arouse  suspicion  and  should  be 
explained  in  the  study  (Baruch,  1999). 

An  interesting  study  of  voters  showed  that  a  mail  survey  with  a  20%  return  rate  predicted 
election  outcomes  better  than  a  phone  survey  with  a  60%  response  rate.  One  explanation  is  that 
the  phone  survey  did  several  follow-ups  to  achieve  its  higher  return  rate.  In  doing  so,  the 
researchers  increasingly  added  respondents  who  were  generally  non-participants.  Not  only  did 
the  subjects  not  respond  to  the  survey  the  first  time  around,  but  also  they  did  not  vote,  and  thus 
were  less  aware  of  the  election  issues  (Krosnick,  1999).  Therefore,  adding  these  non-participants 
to  the  respondent  pool  through  successive  follow-ups  actually  diluted  the  quality  of  the  survey 
data. 


This  problem  of  reluctant  respondents  can  also  arise  with  administered  surveys,  where 
the  surveyor  distributes  the  survey  to  a  group,  is  present  while  they  complete  it,  and  then  collects 
the  surveys.  Theoretically,  this  is  a  100%  return  rate,  but  in  reality  a  percentage  of  respondents 
probably  would  not  have  answered  the  survey,  if  they  really  had  a  choice;  e.g.,  it  had  been 
mailed  to  them.  By  involuntarily  forcing  these  reluctant  individuals  to  complete  the  survey,  the 
data  may  be  skewed  by  these  respondents  purposefully  distorting  their  answers  (Baruch,  1999). 

Theoretical  Explanation  of  Survey  Return  Rates 

There  are  two  basic  theoretical  approaches  explaining  survey  return  rates.  One  examines 
the  theoretical  reasons  that  some  variables  tend  to  increase  return  rates;  i.e.,  ways  to  increase 
response.  The  other  theoretical  perspective  looks  at  why  individuals  do  not  complete  surveys; 
i.e.,  it  explains  nonresponse  behavior. 

Social  Exchange  Theory 

Social  exchange  theory  can  explain  several  of  the  factors  indicated  above  that  affect 
return  rate  (Dillman,  2000;  Martin,  Duncan,  Powers,  &  Sawyer,  1989).  Basically,  exchange 
theory  posits  that  individuals  tend  to  do  things  that  maximize  the  rewards  and  minimize  the  costs 
in  a  transaction.  In  the  survey  situation,  anything  that  maximizes  the  rewards  to  the  individual, 
like  discovering  new  information  from  a  salient  survey  or  even  finding  an  incentive  in  the 
package,  should  motivate  the  individual  to  complete  the  survey  and  return  it. 

Conversely,  anything  that  increases  the  cost  to  the  individual  may  decrease  the  desire  to 
compete  the  survey.  Thus,  longer  surveys  may  be  perceived  as  too  arduous  (too  high  a  cost)  to 
complete,  and  certified  mail  follow-ups  may  require  too  much  effort  to  track  down.  In  addition, 
respondents  may  perceive  hard  to  read  items  and  difficult  to  answer  questions  as  costs  in  terms 
of  time  and  effort  (Fowler,  1993). 
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Naval  Equal  Opportunity/Sexual  Harassment  Survey  (NEOSH)  of  Nonresponse 

Perhaps  the  most  comprehensive  study  of  response  rate  in  EO  surveys  was  recently 
completed  by  the  Navy  (Rosenfeld  et  al.,  2002).  From  1989  to  1999  the  response  rate  on  the 
Navy  Equal  Opportunity/Sexual  Harassment  Survey  (NEOSH)  had  dropped  from  60%  to  30%  - 
a  sizeable  decrease.  Similarly,  the  Navy  Personnel  Survey  (NPS),  from  the  period  1990  to  2000 
dropped  from  52%  to  33%.  These  trends  led  the  Navy  to  devise  a  study  to  examine  factors  that 
might  increase  falling  return  rates. 

A  sub  sample  of  the  1999  NEOSH  sample  was  contacted  by  phone  to  participate  in  a  mail 
survey  and  phone  survey.  The  incentive  was  a  $5  phone  card.  The  return  rate  of  this  mail 
survey  was  39%.  The  results  showed  that  contrary  to  the  prevailing  opinion  that  individuals 
today  receive  too  many  surveys  (Baruch,  1999),  enlisted  personnel  and  officers  did  not  think 
they  were  over-surveyed.  The  mean  number  of  surveys  these  Naval  personnel  received  for  the 
previous  12  months  was  only  two.  In  addition,  a  pair  of  factors  was  identified  as  most  likely  to 
depress  return  rate:  belief  that  surveys  had  no  impact  on  Navy  policy  and  survey  length.  The 
study  also  recommended  that  the  Navy  provide  survey  feedback  to  respondents,  which  in  part 
would  show  the  importance  (salience)  of  the  surveys,  involve  commands  in  surveys  to  show  their 
importance,  shorten  surveys  (they  recommended  about  8  pages),  make  surveys  more  interesting 
by  surveying  current  “hot”  topics  like  religious  discrimination  and  the  impact  of  9/1 1  on  EO,  and 
finally  allow  incentives  in  military  surveys. 

Length 


One  of  the  strongest  findings  of  the  NEOSH  nonresponse  study  was  that  respondents 
thought  the  survey  was  too  long  (Rosenfeld  et  al.,  2002).  There  are  other  indicators  as  well  that 
EO  surveys  are  too  lengthy.  Turning  back  to  Table  1  of  this  report,  the  reader  can  see  that  as 
length  of  EO  surveys  increases,  return  rate  tends  to  drop.  In  fact,  focusing  upon  the  eight 
standardized  EO  surveys  in  Table  1  (MEOCS,  Armed  Forces  EO  Survey,  Mixed  Gender  Opinion 
Questionnaire,  Sexual  Experiences  Questionnaire,  SLEOCS,  MEOCS-LITE,  and  DoD  Sexual 
Harassment  Survey  Forms  A  and  B),  number  of  items  in  the  survey  correlates  negatively  with 
return  rate  (-0.50).  While  this  finding  is  not  statistically  significant  for  the  small  number  of 
surveys  sampled  (df=6),  it  does  show  that  EO  survey  length  and  return  rate  are  related. 

Researchers  are  currently  looking  into  ways  of  shortening  EO  surveys.  For  example, 
item  response  theory  was  used  to  shorten  the  Sexual  Experiences  Questionnaire  from  23  to  16 
items  -  a  31%  reduction  without  loss  of  psychometric  power.  Item  response  theory  allows 
deletion  of  survey  items  that  contribute  less  to  survey  scales  while  still  retaining  informational 
content  and  high  precision  measurement  in  the  remaining  items  (Stark,  Chernyshenko,  Lancaster, 
Drasgow,  &  Fitzgerald,  2002).  Similarly,  efforts  to  reduce  the  size  of  the  various  versions  of  the 
MEOCS  using  item  response  theory  are  ongoing  (e.g.,  Truhon,  2000). 

Survey  Salience 

The  civilian  literature  has  indicated  that  survey  salience  -  its  importance  to  and  impact 
upon  the  individual  -  is  a  factor  in  the  individual’s  choice  to  respond  to  the  survey  (Dillman, 
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Survey  Nonresponse  Theory 

Nonresponse  theory  examines  organizational  factors  that  affect  survey  completion  by 
individuals  (Tomaskovic-Devey,  Leiter,  &  Thompson,  1994).  According  to  this  theory,  there  are 
three  basic  factors  that  influence  nonresponse:  authority  to  respond  (i.e.,  the  individual  has 
formal  or  informal  authority  to  respond  to  a  survey  request),  capacity  to  respond  (the  individual 
has  access  to  the  right  information),  and  motivation  to  respond  (individual  and  organization- 
based  motives  to  respond). 

In  addition,  several  organizational  variables  affect  these  three  factors.  Size  is  one 
variable.  As  organizations  increase  in  size,  complexity  increases.  Authority  to  respond  then 
depends  upon  the  hierarchy  and  decision-making  structure  of  the  organization.  Division  of  labor 
in  larger  organizations  compartmentalizes  information,  and  thus  reduces  capacity  to  respond. 
Large  organizational  hierarchies  also  fragment  motivation  to  respond  among  loyalties  to  the 
organization,  division,  and  department. 

An  organization  with  more  than  1,000  members  tends  to  have  a  12%  lower  response  rate 
than  smaller  organizations  with  10  to  25  members.  The  larger  organizations  have  more  detached 
employees,  who  do  not  identify  readily  with  organizational  goals  that  may  be  reflected  in 
organizational  surveys.  On  the  other  hand,  members  of  small  organizations  may  have  more 
direct  access  to  information  and  be  more  highly  committed  to  organizational  goals  in  surveying 
(be  more  motivated). 

Another  variable  is  degree  of  decision-making  centralization.  Highly  centralized 
organizations  where  most  important  decisions  are  made  at  the  top  will  have  individuals  who  are 
not  authorized  to  respond  and  have  little  information  nor  inclination  (motivation)  to  respond. 

Still  another  variable  is  interaction  with  the  environment.  Organizations,  like  service 
organizations,  that  actively  interact  with  their  environment,  particularly  clients,  will  have  more 
access  to  information  (greater  capacity  to  respond)  as  well  as  a  stronger  motivation  to  respond  to 
outside  surveys.  On  the  other  hand,  organizations  insulated  from  their  environments,  like  large 
bureaucracies,  may  have  neither  the  capacity,  nor  motivation,  to  respond  to  outside  surveys. 

There  are  several  parallels  here  to  military  organizations.  Military  organizations  tend  to 
be  large,  bureaucratic,  centralized  in  terms  of  decision-making,  and  fragmented  as  to  individual 
access  to  organizational  information.  Therefore,  according  to  nonresponse  theory,  we  would 
predict  lower  response  rates,  particularly  in  larger,  highly  centralized  military  units,  which  we 
tend  to  see  in  unit  surveys,  such  as  the  Military  Equal  Opportunity  Climate  Survey  (MEOCS)  in 
Table  1  (Landis,  Dansby,  &  Faley,  1993). 

Review  of  the  Military  Literature  on  Survey  Return  Rates 

While  the  military  literature  on  return  rates  is  not  as  extensive  as  the  civilian  literature, 
several  findings  stand  out. 
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2000;  Edwards  et  al.,  1997).  The  Navy  return  rate  survey  described  above,  showed  that  salience 
can  be  increased  by  involving  commands  more  and  surveying  topics  important  to  respondents 
(Rosenfeld  et  al.,  2002). 


Salience  can  also  be  increased  by  emphasizing  the  importance  of  respondent  involvement 
in  the  survey  to  the  organization  (Edwards  et  al.,  1997).  A  personal  military  anecdote  illustrates 
this  concept.  When  we  were  administering  military  surveys  at  the  Air  Force  Human  Resources 
Laboratory  in  the  early  1970s,  we  were  searching  for  means  of  increasing  the  return  rates  of 
surveys  we  were  sending  to  Air  Force  personnel.  One  day  I  discovered  an  ink  stamp  with  the 
imprint  “For  Your  Eyes  Only”.  We  decided  to  experiment  and  stamp  the  next  batch  of  outgoing 
surveys  with  that  phrase.  To  our  amazement,  the  return  rate  on  that  batch  was  over  90%. 
Evidently,  this  stamp  had  dramatically  increased  the  importance  of  the  survey  in  the  eyes  of 
potential  respondents. 

Demographics 

Gender .  Navy  women  are  more  apt  to  return  surveys  than  Navy  men  (Thomas  & 

Thomas,  1993).  One  important  factor  appears  to  be  the  content  of  the  survey  -  a  primary  factor 
in  survey  salience  (Dillman,  2000).  Questionnaires,  which  evaluate  areas  important  to  women, 
such  as  sexual  harassment,  tend  to  have  high  return  rates.  Culbertson  and  Rosenfeld  (1993)  offer 
a  number  of  ideas  for  improving  a  military  sexual  harassment  survey,  and  thus  improving  return 
rates:  design  the  survey  with  the  using  organization  in  mind,  include  a  definition  of  sexual 
harassment,  assess  the  sexual  harassment  climate  of  the  organization,  ask  about  types  and 
frequencies  of  harassing  behaviors,  and  assess  consequences  to  the  individuals  and  the 
organization. 

Pay  Grade.  Higher  ranking  Navy  noncommissioned  officers  (NCOs)  and  Navy  officers 
tend  to  have  high  return  rates,  at  times  exceeding  70%,  while  lower  ranking  Navy  enlisted 
personnel  (e.g.,  E2s)  have  low  rates  of  response.  These  low  rates  of  response  occur  for  both 
male  and  female  sailors  (Thomas  &  Thomas,  1993). 

Summary  of  Factors  Increasing  Survey  Return  Rates 

A  number  of  meta-analyses,  as  well  as  specific  studies  of  survey  return  rates,  propose 
several  ways  to  increase  return  rates. 

1.  Multiple  Contacts 

The  single  most  important  factor  in  increasing  response  rates  is  multiple  contacts  with  the 
respondents  (Edwards  et  al.,  1997).  A  pre-notification  can  create  expectation  in  the 
respondents  of  an  important  survey  to  follow  (Dillman,  2000).  Most  importantly,  follow¬ 
ups  do  increase  return  rates.  The  format  of  the  follow-up  (letter,  phone,  or  e-mail)  is  less 
important  than  the  attempt  to  contact  the  unresponsive  individual  (Edwards  et  al.,  1997). 
Survey  practitioners  recommend  a  multi-contact  system:  pre-notification  letter,  the 
questionnaire,  a  thank-you  postcard  a  week  to  10  days  later,  a  replacement  questionnaire 
two  to  four  weeks  later,  and  a  final  contact  by  phone  or  FedEx  two  to  four  weeks  after 
that  (Dillman,  2000;  Fowler,  1993).  Research  shows,  however,  that  multiple  follow-ups 
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by  e-mail  on  Web-based  surveys  can  actually  decrease  return  rate  by  overwhelming  the 
individual  (Cook  et  al.,  2000). 

2.  User-Friendly  Salient  Survey 

In  terms  of  social  exchange  theory,  anything  that  can  increase  the  rewards  and  decrease 
the  costs  of  completing  the  survey  will  increase  the  return  rate  (Dillman,  2000).  A  user- 
friendly  survey  emphasizes  simple  language,  well-defined  terms,  and  other  devices  for 
easing  its  completion.  Moreover,  subjects  are  more  apt  to  complete  a  survey  they  find  as 
salient  (important  to  themselves)  (Dillman,  2000;  Edwards  et  al.,  1997).  For  example, 
women  may  be  more  apt  to  complete  a  survey  on  unique  women’s  issues,  because  many 
have  experience  with  these  problems  (Thomas  &  Thomas,  1993).  Respondents  in  general 
are  more  apt  to  complete  surveys  that  cover  information  important  to  them  (Rosenfeld  et 
al.,  2002). 

3.  First-class  Postage 

First-class  postage,  especially  specialized  stamps  like  commemorative  issues  with 
multiple  colors,  may  attract  the  attention  of  respondents  as  they  open  the  package 
(Armstrong  &  Lusk,  1987).  They  may  believe  that  costlier  first-class  postage  indicates 
the  importance  of  the  survey. 


4.  Incentives 

Again,  according  to  exchange  theory,  incentives  in  the  survey  package  may  indicate 
increased  rewards  to  the  respondent  (Dillman,  2000).  While  cash  significantly  increases 
return  rates,  other  material  incentives,  such  as  coupons,  stickers,  small  gifts,  prizes,  and 
phone  cards  can  be  effective  (Arzheimer  &  Klein,  1999;  Church,  1993;  Rosenfeld  et  al., 
2002). 

5.  Shorter  Survey 

Longer  surveys  decrease  return  rates.  The  question  then  becomes  where  is  the  point  at 
which  surveys  become  too  long.  The  arbitrary  number  appears  to  be  four  pages  and 
about  50  items  (Church,  2001;  Edwards  et  al.,  1997).  Surveys  less  than  four  pages  and 
50  items  have  higher  return  rates.  It  is  important  to  note,  however,  that  short  surveys  that 
have  fonts  that  are  too  small,  have  abbreviated  questions,  and  use  undefined  terms  in 
order  to  conserve  space,  may  generate  lower  return  rates  (Dillman,  2000).  The  essential 
point  is  to  balance  survey  length  and  understandability  of  the  survey. 

6.  Organizational  Factors 

Nonresponse  theory  predicts  higher  return  rates  from  individuals  who  work  in  smaller, 
less  bureaucratic,  organizations  with  decentralized  decision  making  and  direct  access  to 
relevant  information  required  to  answer  survey  questions  (Tomaskovic-Devey  et  al., 
1994). 
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Recommendations  for  Increasing  Return  Rates  in  Military  EO  Surveys 

Based  upon  the  above  discussion,  several  recommendations  for  increasing  the  return  rate 
of  military  EO  surveys  follow. 

1 .  Consider  greater  use  of  Web-based  surveys 

The  military  by  its  very  nature  is  a  young  population.  NCOs  must  retire  in  their  30’s  and 
40  s,  and  officers  must  retire  in  their  40’s  and  50’s.  Evidence  shows  that  younger 
populations  prefer  Web-based  information  exchange,  including  Web-based  surveys 
(Cook  et  al.,  2000). 

2.  Use  multiple  contacts  of  respondents 

Multiple  contacts  in  the  form  of  pre-notifications  and  follow-ups  are  the  strongest  factor 
in  increasing  return  rates  (Edwards  et  al.,  1997).  With  Web-based  surveys,  this  becomes 
relatively  easy  through  e-mail.  It  should  be  cautioned,  however,  that  excessive  e-mail 
contacts  could  backfire  by  overwhelming  the  individual  and  actually  decreasing  return 
rates  (Cook  et  al.,  2000). 

3.  Shorten  surveys 

Historically,  EO  surveys  have  been  long.  The  basic  MEOCS,  for  example,  has  124 
items,  and  its  return  rate  is  low.  A  more  optimal  size  would  appear  to  be  about  half  that 
size  -  about  60  items  (Rosenfeld  et  al.,  2002).  Current  attempts  to  convert  the  MEOCS 
into  shorter  and  more  specialized  modules  would  seem  to  be  a  step  in  the  right  direction 
(e.g.,  Kustra,  2002;  Truhon,  2000). 

4.  Consider  unique  incentives 

Traditionally  military  surveyors  assumed  that  military  members  would  respond  through  a 
sense  of  duty.  Given  the  number  and  availability  of  surveys  these  days,  military 
members  are  becoming  more  selective  about  which  surveys  they  are  choosing  to  respond 
to.  Exchange  theory  would  posit  that  increasing  rewards  would  increase  participation 
(Dillman,  2000).  Unique  incentives,  such  as  phone  cards  (Arzheimer  &  Klein,  1999; 
Rosenfeld  et  al.,  2002),  may  be  particularly  attractive  to  military  members  who  may  be 
stationed  far  from  friends  and  loved  ones  and  desire  to  communicate  with  them.  In 
addition,  points  that  can  be  traded  for  desired  goods  and  merchandise  (Sheehan,  2001) 
may  be  an  attractive  means  of  increasing  return  rates  of  Web-based  surveys. 

5.  Consider  more  localized  surveys 

Nonresponse  theory  posits  that  very  large  bureaucratic  organizations  (in  this  case  the 
military)  tend  to  produce  lower  response  rates  (Tomaskovic-Devey  et  al.,  1994).  Large 
bureaucratic  organizations  tend  to  put  hurdles  in  the  way  of  responding  in  terms  of 
greater  difficulty  accessing  information,  rules  restricting  types  of  response,  and  less 
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commitment  on  the  part  of  individuals  that  translates  into  lower  survey  participation. 
Therefore,  the  military  should  consider  more  localized  EO  surveys  at  least  at  the  service 
or  major  command  level,  where  a  greater  degree  of  commitment  may  attach  to  the 
organization.  This  may  have  particular  ramifications  for  the  MEOCS,  which  has 
traditionally  been  administered  as  one  survey  through  differing  types  of  military  units 
(Landis  et  al.,  1993).  One  option  may  involve  eliminating  the  MEOCS  as  a  monolithic, 
DoD-level  survey  instrument  and  replacing  it  with  several  versions  that  are  more  service 
and  major  command  specific. 


Conclusions 

Most  of  what  we  know  about  the  effects  of  diversity  and  about  equal  opportunity  we  have 
learned  through  EO  surveys.  Therefore,  effective  survey  administration,  including  enhanced 
survey  return  rates,  is  critical  for  ensuring  high  quality  data  upon  which  important  policy 
decisions  will  be  made.  The  civilian  and  military  reviews  of  the  literature  reveal  that  several 
factors  can  increase  equal  opportunity  survey  return  rates;  greater  use  of  Web-based  electronic 
surveys,  multiple  contacts,  including  follow-ups,  shorter  surveys,  creative  incentives  for 
returning  surveys,  and  more  localized  surveys. 
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